Full Field Visual-Mid-Infrared Imaging System

ABSTRACT

An apparatus and method for generating images of specimens is disclosed. The apparatus includes an imaging system, controller, and user interface. The imaging system generates a plurality of component images of a specimen, each component image corresponding to a different viewing condition. Each image is represented by an intensity as a function of location on the specimen. The controller stores the component images and generates a compound image from a plurality of the component images. The compound image includes a weighted sum of first and second ones of the component images, the controller displaying the compound image on a display controlled by the controller. The user interface is adapted to control a weighting factor used in generating the weighted sum in response to user input. The controller redisplays the compound image after the weighting factor is changed in response to user input.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application is divisional of co-pending U.S. patent applicationSer. No. 15/975,701 filed May 9, 2018, which is a continuation ofco-pending U.S. patent application Ser. No. 14/743,854 filed on Jun. 18,2015, now U.S. Pat. No. 10,003,754, said patent applications beingincorporated by reference herein.

BACKGROUND

Quantum cascade lasers provide a tunable mid-infrared (MIR) light sourcethat can be used for spectroscopic measurements and images. Manychemical components of interest have molecular vibrations that areexcited in the MIR region of the optical spectrum, which spanswavelengths between 5 to 25 microns. Hence, measuring the absorption ofMIR light at various locations on a sample can provide usefulinformation about the chemistry of the sample as a function of positionon the sample.

SUMMARY

The present invention includes an apparatus and method for generatingimages of specimens. The apparatus includes an imaging system,controller, and user interface. The imaging system generates a pluralityof component images of a specimen, each component image corresponding toa different viewing condition. Each image is represented by an intensityas a function of locations on the specimen. The controller stores thecomponent images and generates a compound image from a plurality of thecomponent images. The compound image includes a weighted sum of firstand second ones of the component images, the controller displaying thecompound image on a display controlled by the controller. The userinterface is adapted to control a weighting factor used in generatingthe weighted sum in response to user input. The controller redisplaysthe compound image after the weighting factor is changed in response touser input.

In one aspect of the invention, one of the component images is stored incompressed format by the controller.

In another aspect of invention, the first component image includes aregion having a different spatial resolution than the second componentimage.

In another aspect of invention, the first component image corresponds tothe specimen being illuminated by a different wavelength of light thanthe second component image. The first component image can includeintensity values for the entire specimen.

In another aspect of invention, the second component image does not haveintensity values for points on the specimen at which the first componentimage has intensity values.

In another aspect of invention, the second component image includesregions having different spatial resolutions.

In another aspect of invention, the controller is adapted to receiveinformation specifying a viewing condition and a scan region on thespecimen, the controller causing the imaging system to generate an imageof the specimen using the viewing condition in the scan region. Theviewing condition can include an illumination wavelength, and if noother component image has a viewing condition with that illuminationwavelength, the controller defines a new component image for thegenerated image. The scan region can include a region of an existingcomponent image, and the generated imaging replaces intensity values atcorresponding locations in the existing component image.

In another aspect of invention, each image is represented by avector-valued function as a function of locations on the specimen, theintensity as a function of location on the specimen being one componentof that vector-valued function. The controller can convert the compoundimage to a colored image, the color components depending on thevector-valued components.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates one embodiment of a scanning MIR imaging system.

FIG. 2 illustrates an embodiment of an imaging system in which the lensassembly moves.

FIG. 3 illustrates an embodiment of a MIR imaging system that combines avisual imaging system with a MIR imaging system.

FIG. 4 illustrates a cross-section through a y-plane in thethree-dimensional image.

FIG. 5 illustrates a multi-resolution image according to one embodimentof the present invention.

FIG. 6 illustrates a display in which the visible image and one of theMIR images are chosen for display.

DETAILED DESCRIPTION

The manner in which the present invention provides its advantages can bemore easily understood in the context of a scanning MIR imaging systemthat also provides an image in the visual spectrum of the specimen beingscanned. Refer now to FIG. 1 which illustrates one embodiment of ascanning MIR imaging system. Imaging system 10 includes a quantumcascade laser 11 that generates a collimated light beam 18 having anarrow band of wavelengths in the MIR. In one aspect of the invention,quantum cascade laser 11 is a quantum cascade laser having a tunablewavelength that is under the control of a controller 19. Collimatedlight beam 18 is split into two beams by a partially reflecting mirror12. Light beam 18 a is directed to a lens 15 that focuses that beam ontoa specimen 16 that is mounted on xy-stage 17 that can position specimen16 relative to the focal point of lens 15. Light that is reflected backfrom specimen 16 is collimated into a second beam that has a diameterdetermined by the aperture of lens 15 and returns to partiallyreflecting mirror 12 along the same path as light beam 18 a. While thefirst and second beams are shown as having the same cross-section inFIG. 1, it is to be understood that the second beam could have adifferent cross-section than the first beam. A portion of the secondbeam is transmitted through partially reflecting mirror 12 and impingeson a first light detector 13 as shown at 18 b. Light detector 13generates a signal related to the intensity of light in beam 18 b.Controller 19 computes an image as a function of position on specimen 16by moving specimen 16 relative to the focal point of lens 15 usingxy-stage 17.

Controller 19 also monitors the beam intensity of the light incollimated light beam 18 using a second light detector 14 that receivesa portion of the light generated by quantum cascade laser 11 throughpartially reflecting mirror 12. Quantum cascade laser 11 is typically apulsed source. The intensity of light from pulse to pulse can varysignificantly, and hence, the pixels of the image are corrected for thevariation in intensity by dividing the intensity measured by lightdetector 13 by the intensity measured by light detector 14. In addition,since the light intensity from quantum cascade laser 11 is zero betweenpulses, controller 19 only sums the ratio of intensities from lightdetectors 13 and 14 during those times at which the output of lightdetector 14 is greater than some predetermined threshold. This aspect ofthe present invention improves the signal-to-noise ratio of theresultant image, since measurements between pulses contribute onlynoise, which is removed by not using measurements between pulses.

The potential resolution in terms of scan lines of this type of imagingsystem is very large. The maximum specimen size is set by the maximumtravel of the specimen stage. Stages that move tens of centimeters areeasily accommodated. The spot size on the specimen is limited by opticsto one or two wavelengths of the MIR light. Hence, a spot size of 30microns is achievable. Thus, to scan a large specimen at the highestresolution requires thousands of scan lines. Accordingly, reducing thescanning time is of utmost importance.

In the above described embodiments, the stage moves the sample in twodimensions. However, the stage has a significant mass, and hence, thespeed at which the sample is imaged is limited by the motion of thestage. In embodiments in which rapid imaging time is important,embodiments in which the specimen is scanned in one direction by movinglens 15 are preferred. Refer now to FIG. 2, which illustrates anembodiment of an imaging system in which the lens assembly moves. Inimaging system 30, the stage assembly is divided into two components.Component 31 includes focusing lens 55 and is moveable in a directionshown at 32 such that a single line of the image is generated with eachpass of component 31. Since focusing lens 55 and mirror 56 have a massthat is small compared to component 57, component 31 can be moved withmuch greater speed. In one embodiment, component 31 is mounted on a railand moved in a manner analogous to a print head on an inkjet printer.The second component of the stage assembly is shown at 57. Component 57includes the mounting mechanism for the specimen being scanned and movesin a direction 33 that is orthogonal to direction 32. Since component 57only needs to move once per scan line, the slower speed of motionassociated with the more massive component 57 is acceptable. Controller39 provides control function analogous to controller 19 shown in FIG.19.

However, even with this improved scanning speed, a complete scan of alarge specimen could require hours, particularly, if scanning at anumber of different MIR wavelengths is required. Accordingly, anarrangement in which the specimen can be imaged in the visiblewavelengths followed by the user selecting specific areas to be scannedin the MIR would be advantageous.

Refer now to FIG. 3, which illustrates an embodiment of a MIR imagingsystem that combines a visual imaging system with a MIR imaging system.Imaging system 60 includes a MIR imaging system 70 and a visible lightimaging system 80 having an imaging array 81 that is responsive to lightin the visible wavelengths. Both systems are under the control ofcontroller 61. The two imaging systems share a stage 66 that allows aspecimen 65 to be moved between the imaging systems such that an areadefined using visible light imaging system 80 can be positioned forscanning using MIR imaging system 70. MIR imaging system 70 includes ascanning head 73 that moves on a rail 72 under the control of controller61 in a manner analogous to that described above with respect to imagingsystem 30 shown in FIG. 3. Stage 66 allows the specimen to be moved in adirection perpendicular to the direction of travel of scanning head 73so that a two-dimensional image can be generated. To simplify thedrawing, the laser and associated optics in MIR imaging system 70 areshown as a single block 71.

In practice, a user places specimen 65 in position under visible lightimaging system 80 and indicates which portion of specimen 65 is to bescanned using a user interface 62 and display 63. The user can indicatethe desired area using a point device or similar apparatus. The visibleimage is generated using an objective lens that provides the desiredlevel of magnification. Controller 61 then computes the distancespecimen 65 must be moved to be properly aligned with MIR imaging system70. Specimen 65 is then moved and scanned as described above.

Managing and viewing the large amount of data generated in thesemultiple scans presents significant challenges. In general, there willbe one visible image of the entire specimen and a number of small areascans at various locations on the specimen. In addition, some areas ofinterest can have multiple scans at different wavelengths. Finally,different sub-images can have different spatial resolutions.

The present invention utilizes a three-dimensional image structure tomanage this large amount of information. For the purposes of the presentdiscussion, any given image is defined as an intensity function on anx-y plane. For example, an image in the visual wavelength could berepresented by an intensity of reflected light as a function of thestage position (x,y) of the point being measured. In the z-direction,the image is divided into “layers”. Refer now to FIG. 4, which is across-section through a y-plane in the three-dimensional image. Eachlayer is an image associated with a particular wavelength of MIR lightor visible light. Exemplary planes are shown at 101-104. To simplify thefollowing discussion, it will be assumed that the visual image is amonochrome image located at z=0, i.e., plane 101. However, the visualimage could be a “color” image, in which case, the image pixels would bevectors defined on the x-y plane corresponding to the visual image. Tosimplify the present discussion, it will be assumed that all images arescalar valued intensities as a function of (x,y). The extension of theinvention to a vector-valued image pixel will be discussed in moredetail below.

In general, the (x,y) coordinates are not continuous values, but ratherdigital values that represent locations on the sample stage. The finestincrements in x,y are preferably set to be the spacing of the highestresolution measurements in x,y. That is, the stage is moved by adistance of 1 unit in x or y between successive measurements. Ingeneral, the resolution at which any given image or portion of an imageis taken can vary from region to region in the image or change withwavelength. For example, in one region of interest on the sample, aregion could be scanned with the smallest possible spot size andincrementing the stage by a distance of one count in x or y. In anotherregion, samples could be taken by incrementing the stage by two countsbetween measurements to improve the speed of the measurements.

When displaying an image, the resolution with which the data was takenis reflected in the pixel size on the image. There is a one-to-onerelationship between the pixels and the coordinates on the sample stage.At the highest resolution, each pixel represents a 1×1 square on thesample stage and the intensity of that pixel is determined by themeasured reflectivity of the corresponding point on the sample. Ifmeasurements are only taken every two pixels, then each measurementrepresents an estimate of the intensity at the four pixels that includethe measurement point. The intensities of these four pixels are set tothe measured value.

Refer now to FIG. 5, which illustrates a multi-resolution imageaccording to one embodiment of the present invention. In this example,most of the specimen is scanned by taking points every five counts inthe sample stage position. Hence, these pixels are 5×5 squares such aspixel 113. A small portion of the image was scanned at the resolution ofone measurement for each count in the sample stage position. Thesepixels are shown at 111, and consist of 1×1 squares.

It should be noted that in embodiments that utilize a fast moving lensassembly for scanning in the x direction, no time improvement isobtained by taking data at coarser intervals in the x-direction. Theincreased speed is obtained by incrementing the stage position in they-direction by multiple counts between scans. For example, the stagecould be incremented by two counts in the y-direction between x scansthat take data every count in the x-direction. In this case, the pixelsare represented by 1×2 rectangles that have an intensity determined bythe values measured at x position. A region with such pixels is shown at112.

In general, the visible image will be recorded over the entire samplestage. The other images, however, may only be recorded over smallerareas. In addition, those smaller areas may differ from image to image.Hence, keeping track of the locations of the portions of the otherimages that have been measured presents challenges. In one aspect of theinvention, the controller provides an image display in which multipleimages are viewed superimposed on one another with the upper imagesbeing partially transparent. In essence, the user views the image stackfrom a position above the stack on the z-axis with the individual imagesassigned a transparency value, so that features in the lower images canbe seen through the upper images. In the simplest case, only two imagesare visible at once, i.e., the other images are assigned a transparencythat renders those images invisible.

Refer now to FIG. 6, which illustrates a display in which the visibleimage and one of the MIR images are chosen for display. In this example,the sample consists of a capsule such as an aspirin tablet. Thedistribution of the aspirin in the tablet can be viewed in the MIR,since the aspirin absorbs in the MIR and the filler material does not.The visual image is shown at 141. A number of MIR images of portions ofthe specimen are shown at 142-146. The regions of the MIR image thathave not been scanned are completely transparent. While the MIR scannedareas in this example are sufficiently opaque to block the underlyingvisual image, it is to be understood that the transparency of theregions shown at 142-146 could be set to allow the underlying visualimage to be viewed through the MIR imaged regions.

In one aspect of the invention, the transparency of the upper image canbe continuously set with a control such as a slider. The user can startwith the slider set such that the upper image is completely transparent,and hence, the user is viewing only the underlying visible image. As theuser increases the opaqueness of the upper layer, the upper layerbecomes visible and the user can compare features in the two images. Asthe user further increases the opaqueness of the upper layer, theunderlying image becomes blocked by the regions of the upper image thathave been scanned.

In the above example, the bottom layer was the visible image; however,the user can define any two images as the upper and lower layers forviewing. Hence, the user can view portions of the sample at twodifferent MIR scan wavelengths and pan from one to the other to view thechanges in the images with wavelength.

In one aspect of the invention, the individual images are assigneddistinct colors. Hence, the areas in which the images overlap will beseen in a distinct color that depends on the transparency and colorschosen for the individual images. For example, red could be assigned forthe first layer and blue for the second layer. Features that are presentin both images would then be seen in various shades of purple.

The above described examples utilize two layers for viewing. However,embodiments in which the user defines a transparency level and color foreach layer in the stack of images can also be constructed.

A display according to the present invention can be used to direct theimaging system during scanning. The user can view one of the images andchoose a region to be scanned. The user then enters the scanningparameters such as the resolution and wavelength of light to be used.The imaging system then scans that region and enters the new data intothe array that stores the three-dimensional image. If the wavelengthchosen matches a wavelength for which an image plane has already beendefined, the new data is entered on that plane. If other imaging data isalready present on that plane at the corresponding x,y values, the newdata replaces the existing data and is entered at the new resolution.For example, the user could choose a region in which a coarse resolutionscan had already been made to provide data at a finer resolution. Thenew fine resolution data would then be entered to replace the previouslyscanned data.

The display system of the present invention can be applied to a numberof different imaging systems in addition to those described above. Forexample, one type of system for generating images in the MIR can beviewed as a conventional microscope with a MIR light source and anoptical system that images the illuminated sample onto an array of MIRdetectors. A MIR light source based on a quantum cascade laser providesa tunable MIR light source. There are several problems with such imagingsystems. In such systems, a high resolution image of a large specimenmust be generated by “stitching” together a number of smaller highresolution images. The specimen can be scanned at a low resolution toachieve an image of an entire large specimen. The magnification is thenchanged and smaller high resolution images are taken. These images maybe combined to provide a high resolution image of the specimen or mayremain as isolated high resolution areas. In addition, the specimen orparts thereof, can be scanned at different wavelengths to provideadditional images. Hence, a display problem that is analogous to thatdescribed above is also encountered in this type of imaging system.

Accordingly, the display system of the present invention is applicableto any scanning system in which a number of images of the same specimenare generated under different “viewing conditions” without altering thespecimen. A viewing condition is defined to be the illuminationconditions of the specimen such as wavelength and intensity of incidentlight and the measurement conditions for the light absorbed or reflectedby the specimen. In one set of viewing conditions, the illuminationwavelength differs from image to image. In a second set of viewingconditions, two or more images are taken with the same incident lightwavelength, but the light leaving the specimen is viewed throughdifferent forms of filters. For example, the illuminating light could belinearly polarized in a first direction and a polarization filter usedto view the light leaving the specimen. Such systems are useful inseparating light that is diffusely reflected from the specimen fromlight that is specularly reflected from the specimen. In one aspect ofthe invention, the specimen is not removed from the scanning apparatusbetween images.

In this general case, each image is stored on a different image plane ina manner analogous to that discussed above. Each image will be referredto as a component image in the following discussion. The componentimages are registered with respect to one another. That is, the (x,y)coordinates of the intensity values in each component image are allreferenced to the (x,y) coordinates of the specimen. This allows aplurality of component images having different viewing conditions to becompared by overlapping the images at different transparencies and/orpanning from one image to another. For the purposes of the presentdiscussion, a viewing condition is defined to be the illuminationwavelength of the specimen, the resolution of the image generated, thesample points on the specimen at which the image intensity is actuallymeasured, and any filters or transformations used to process the datacollected at those sample points to generate an intensity as a functionof position on the specimen.

The individual component images can have regions of different spatialresolution within the component image as discussed above. In addition,the individual component images can be incomplete in that only part ofthe specimen was scanned to generate that image. It is advantageous tohave one component image that is a complete image of the specimen underone viewing condition to provide a reference that allows the othercomponent images to be placed in geometric context with each other. Inthe above examples, that reference image was a component image generatedby using light in the visible range of wavelengths. However, otherviewing conditions could be utilized for the reference image.

The controller generates a compound image by forming a weighted sum oftwo or more component images. The weight factors used to combine thecomponent images are equivalent to the transparency factor discussedabove. A small weighting factor renders the corresponding componentimage more transparent than a larger weighting factor. The compoundimage C(x,y) for two component images, I₁(x,y) and I₂(x,y) is given byC(x,y)=w₁ I₁(x,y)+w₂ I₂(x,y), where w₁ and w₂ are the weight factors.The sum of the weight factors is preferably one to prevent changes inoverall intensity of the compound image as the weight factors arechanged.

To combine two component images, there must be corresponding intensityvalues in each component image. As noted above, a component image caninclude pixels of different sizes. For example, if an intensity is onlymeasured at every fifth point on the specimen, there will not beintensity values at the other points. The present invention fills in themissing points by replicating the measured values at each of the missingpoints.

In addition, the component image may only include data in a limitedregion of the specimen area. In one aspect of the invention, when animage plane is defined, the component image is initialized to intensityvalues of zero. Each component image is defined to have storagelocations for each possible (x,y) coordinate on the specimen. As data isgenerated for various regions in that component image, the new datavalues replace the corresponding zeros. The regions that are notmeasured remain zero, and hence, appear completely transparent when thecompound image is generated.

To save memory, the image plane may be stored in a compressed formatrather than providing a two dimensional array of pixels in memory. Forexample, if only a small fraction of the image plane is non-zero,schemes in which the coordinates of each of the small non-zero pixelsare stored together with the intensity values corresponding to thosepixels. If the image plane has rectangular islands of measured pixels,the coordinate of a reference point in each island can be storedtogether with the small measured image. When a pixel value is required,the compressed image can be decompressed to provide the desired pixelvalue. If the coordinate of the pixel is not in one of the non-zeroregions, or “islands”, a value of zero is returned. Schemes that detectlong runs of a particular pixel value such as a zero can also be used tocompress the image plane data in sparsely populated image planes.

The above-described embodiments have utilized images comprising anintensity as a function of location on the specimen. However, an imagecan be defined as a set of vector values defined as a function ofposition on the specimen. For example, in an image in the visualwavelengths, each point on the specimen could be characterized by acolor vector having a plurality of components, each componentrepresenting an intensity at a different color.

The controller can likewise generate a compound image in which each(x,y) value on the specimen has a vector associated with that location.In this case, the vector is a plurality of intensity values. Theweighting process is performed on each component of the vector toprovide a weighted vector-valued image. In the case of a three componentvector, the image can be displayed as a colored image. As noted above,individual component images can be assigned colors. Hence, an intensitybased image can be converted to a vector-valued image by multiplyingeach component of a unit color vector by the measured intensity. Thisallows the controller to provide a colored component image.

The above-described embodiments of the present invention have beenprovided to illustrate various aspects of the invention. However, it isto be understood that different aspects of the present invention thatare shown in different specific embodiments can be combined to provideother embodiments of the present invention. In addition, variousmodifications to the present invention will become apparent from theforegoing description and accompanying drawings. Accordingly, thepresent invention is to be limited solely by the scope of the followingclaims.

What is claimed is:
 1. A method for operating a scanning systemcomprising a first scanning station that generates component images of aspecimen at visual wavelengths, a second scanning station that generatescompound images of portions of said specimen at wavelengths that aredifferent from wavelengths used by said first scanning station, a stagethat moves specimen between first and second scanning stations to allowspecimen to be scanned at each scanning station without being removedfrom said scanning system, said method comprising: generating aplurality of component images including a visual image generated withsaid first scanning station, said visual image representing a region ofinterest on said specimen utilizing a visual wavelength, said pluralityof component images further comprising one or more detail componentimages being characterized by different viewing conditions and/orsub-regions of said specimen, said detail component images beinggenerated by said second scanning station, wherein each of said detailcomponent images comprises measured intensity values for one or moresub-regions of said region of interest, each of said component imagescomprising a two-dimensional array of intensity values having oneintensity value for each location in said region of interest, saidintensity value for each location not in one of said sub-regions beingset to a predetermined value; and generating a compound image of aplurality of said component images, said compound image comprising alocation by location weighted sum of said plurality of component images,and displaying said compound image on a display.
 2. The method of claim1 wherein said weighted sum depends on a set of weights provided by auser, said method comprising redisplaying said compound image on saiddisplay each time said set of weights is altered.
 3. The method of claim1 comprising receiving user input with respect to a viewing conditionand area of said specimen to scan; defining a new component image forthat viewing condition if no other component image has that viewingcondition; and scanning said area of said specimen to provide intensityvalues for that new component image.
 4. The method of claim 1 comprisingreceiving user input with respect to a viewing condition and area ofsaid specimen to scan; scanning said area of said specimen to generatereplacement values for locations corresponding to said area of saidspecimen; and if said viewing condition corresponds to an existingcomponent image, replacing intensity values at locations in one of saidcomponent images with said replacement values.